Demand for Real-Time Video
Explosive growth in consumer and business demand for real-time video on mobile and Internet devices has created exciting new commercial opportunities and major new technical challenges. As they pursue the integration of new real-time video capabilities (FIG. 1) for mobile/Internet communication, business collaboration, entertainment, and social networking, device manufacturers, network infrastructure providers, and service provides are struggling to meet customer expectations for higher quality real-time video across a wider range of devices and networks.
Limitations of Broadcast Video Solutions
Today's standard video processing and distribution technologies have been developed to efficiently support one-way video broadcast, not the two-way and multi-party video sharing required for real-time mobile and Internet user interaction. Traditional broadcast industry solutions have proven to be too computationally complex and bandwidth hungry to deliver the device, infrastructure, or bandwidth requirements for commercially scalable real-time mobile/Internet video services.
Device, Network, and Video Fluctuations
Furthermore, the available computational resources on many devices, as well as the delay, jitter, packet loss, and bandwidth congestion over user networks cannot be guaranteed to remain constant during a real-time video/audio communication session. In the absence of any adaptation strategy, both device and network loading can lead to significant degradation in the user experience. An adaptation strategy designed to address network fluctuations but not device loading fluctuations is ineffective, since it is often difficult to distinguish between these two contributors to apparent “lost packets” and other performance degradations. Adaptation to frame-to-frame fluctuations in inherent video characteristics can provide additional performance benefits.
Embodiments of the present invention comprise an all-software Real-time Video Service Platform (RVSP). The RVSP is an end-to-end system solution that enables high-quality real-time two-way and multi-party video communications within the real-world constraints of mobile networks and public Internet connections. The RVSP includes both Client and Server software applications, both of which leverage low-complexity, low-bandwidth, and network-adaptive video processing and communications methods.
The RVSP Client (FIG. 2) integrates all: video and audio encode, decode, and synchronization functions; real-time device and network adaptation; and network signaling, transport, and control protocols, into a single all-software application compatible with smartphone and PC operating systems. The RVSP client application has been designed to accommodate fluctuations in: the internal loading of client devices; external impairments on a variety of different user networks; and inherent video characteristics such as frame-to-frame compressibility and degree of motion.
The RVSP Server (FIG. 3) integrates multiparty connectivity, transcoding, and automated video editing into a single all-software application. The all-software architecture of the RVSP supports flexible deployment across a wide range of network infrastructure, including existing mobile application/media server infrastructure, standard utility server hardware, or in a cloud computing infrastructure. For both peer-to-peer and server-based real-time 2-way video share services and multi-party video conferencing, the RVSP platform reduces both the up-front capital expenditures (CapEx) and on-going operational expenditures (OpEx) compared to existing video platforms that utilize significantly higher bandwidths and require additional specialized video hardware in both the user devices and the network infrastructure.
In order to meet customer expectations for higher quality video across a wider range of devices and networks, mobile operators and other communication service providers worldwide have made significant new investments in IP Multimedia Subsystem (IMS) network infrastructure. By reducing bandwidth consumption and supporting higher concurrent user loading capabilities for a given infrastructure investment and bandwidth allotment in an IMS deployment (FIG. 4), the RVSP provides significant CapEx and OpEx reductions over competing real-time video platforms that require additional specialized video hardware in both the user devices and the network infrastructure.
The RVSP also delivers similar CapEx and OpEx benefits for “over the top” (OTT) and direct-to-subscriber deployments of real-time video services (FIG. 5) using standard utility server hardware or in a cloud computing infrastructure. In these cases, mobile devices communicating via public Internet or corporate networking infrastructure typically do not have access to video quality-of-service (QoS) enhancements in the mobile operator's IMS core. The real-time network adaptation features of the RVSP disclosed here then become critical to delivering a compelling user experience within the real-world constraints of mobile networks and consumer Internet connections.
Video conferencing systems are evolving to enable a more life-like “Telepresence” user experience, in which the quality of the real-time video and audio communications and the physical layout of the meeting rooms are enhanced so that multiple remote parties can experience the look, sound, and feel of all meeting around at the same table. As shown in FIG. 6, multi-user video conferencing systems typically require specially designed meeting rooms with dedicated video cameras, large size video displays, arrays of audio microphones and speakers, and specialized processing equipment for digitizing, compressing, and distributing the multiple video and audio streams over dedicated high-speed data network connections.
For many consumer and business applications, there is a need to extend higher quality multi-party video communications to participants using a wider variety of less-specialized video-enabled electronic devices, including mobile communications devices, laptop computers, PCs, and standard TVs. There is also a need to extend immersive business communications to support a wider range of consumer and professional collaboration and social networking activities.
When it comes to multi-party video communications, users of these less-specialized electronic devices encounter a number of drawbacks in the devices and in the user experience. For example, these devices may have a wide range of video processing capabilities, video display sizes, available connection bandwidths, and available connection quality-of-service (QoS). Furthermore, without the benefit of specially designed meeting rooms, creating a “perceptually pleasant” meeting experience is challenging. Many video conferencing systems rely on a static screen layout in which all participants are presented within an array of equal-sized video “tiles”, even though several participants may be passive listeners throughout much of the meeting and hence contribute very little. These “static” multi-party video default display layouts have many drawbacks, including:                1. All participants are displayed at the same image size, same image quality, and same video frame rate, regardless of their level of participation.        2. Individual participants have no control over the display layout on their own device.        3. A participant with the role of “moderator” cannot “give the floor” to individual participants, as they can in a face-to-face conference setting.        4. Participants cannot choose to focus on one other participant, as they can in a face-to-face conference setting.        
When deployed together, the RVSP Client and Server applications enable multiple participants to simultaneously create and share high-quality video with each other in real-time, with many key aspects of a face-to-face user experience.